Image processing apparatus, image processing method and imaging apparatus

ABSTRACT

A composite image is generated by a method including: detecting at least one in a captured image; judging arrangement of the at least one subject by an arithmetic operation; selecting at least one composition-purpose image at random in accordance with the arrangement of the at least one subject; determining arrangement of the selected at least one composition-purpose image at random so that the at least one composition-purpose image does not overlap with one another and does not overlap with the detected at least one subject; and generating a composite image by combining the captured image with the at least one composition-purpose image.

This application is a divisional to application Ser. No. 12/370,692,filed Feb. 13, 2009, now U.S. Pat. No. 8,249,389 which is based upon andclaims the benefit of priority from the Japanese Patent Application No.2008-33368 filed on Feb. 14, 2008, the entire contents of which areincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Technical Field

The invention relates to an image processing apparatus, an imageprocessing method and an imaging apparatus. Particularly, the inventionrelates to an image processing apparatus, an image processing method andan imaging apparatus for generating a composite image by combining acaptured image with a composition-purpose image and displaying thecomposite image.

2. Description of the Related Art

For example, JP 2004-248037 A, JP 2003-78818 A, JP 2004-153345 A and JP2005-157762 A describe techniques capable of generating a compositeimage by combining a captured image with a composition-purpose image atthe time of image capturing.

JP 2004-248037 A describes transmitting a location of image capturing toa server at the time of image capturing, receiving from the server clipart images selected at random from additional images provided for eachimage capturing area, and generating a composite image. Since differencecomposite images are generated for respective image capturings even inthe same image capturing area, surprise and amusement can be provided toa user.

JP 2003-78818 A describes automatically detecting a position of asubject and displaying a ticker (a character string or an explanatorygraphic for indicating a name of the subject, description of thesubject, etc.) on a screen in accordance with motion of the subject.

JP 2004-153345 A describes that a template image selected by a cameramanis displayed while being superposed on a through image and that when thecameraman determines a composition and shoots, a shot image combinedwith the template image and a shot image not combined with the templateimage are stored.

JP 2005-157762 A describes extracting a specific object from a capturedimage, generating a composite image in which the extracted object isreplaced by an exchangeable object registered in advance, and outputtingthe composite image.

Incidentally, it was impossible to select a composition-purpose image(s)to be adapted for arrangement (position, size and direction) of asubject, the number of subjects and a positional relation betweensubjects. Moreover, it was impossible to determine arrangement of acomposition-purpose image(s) in accordance with the position of asubject and/or the positional relation between subjects. For thisreason, there was a possibility that a captured image would be combinedwith a composition-purpose image that is unmatched with the scene of thecaptured image or a composition-purpose image would overlap with asubject image or another composition-purpose image.

SUMMARY OF THE INVENTION

The invention has been made in consideration of such circumstances. Anobject of the invention is to provide an image processing apparatus, animage processing method and an imaging apparatus capable of generating acomposite image matched with an arrangement scene by combining acaptured image with a composition-purpose image(s) based on a subject(s)in the captured image.

The above objection of the invention is achieved by the followingconfigurations.

[1] According to an aspect of the invention, an image processingapparatus includes a subject detection unit, a subject position analysisunit, an image selection unit, an image arrangement unit and an imagecomposition unit. The subject detection unit detects at least onesubject in a captured image. The subject position analysis unit judgesarrangement of the at least one subject in the captured image by anarithmetic operation. The image selection unit selects at least onecomposition-purpose image at random in accordance with the arrangementof the at least one subject. The image arrangement unit determinesarrangement of the selected at least one composition-purpose image atrandom so that the at least one composition-purpose image does notoverlap with one another and does not overlap with the detected at leastone subject. The image composition unit generates a composite image bycombining the captured image with the at least one composition-purposeimage.

[2] In the image processing apparatus of [1], the image arrangement unitmay select a size for each of the at least one composition-purpose imageto be arranged, at random from an image size range that is set inadvance.

[3] In the image processing apparatus of any one of [1] to [2], theimage arrangement unit may select an inclination for each of the atleast one composition-purpose images to be arranged, at random from animage inclination range that is set in advance.

[4] In the image processing apparatus of any one of [1] to [3], when thenumber of detected subject is two or more, the image arrangement unitmay select the at least one composition-purpose image based on anarrangement relation between the detected subjects.

[5] In the image processing apparatus of [4], the image selection unitmay select the at least one composition-purpose image based on eachdistance between the subjects.

[6] In the image processing apparatus of any one of [1] to [5], whereinwhen the image composition unit generates the composite image from eachof continuously captured images, if arrangement of the at least onesubject in a current captured image is same as that of the at least onesubject in a previous captured image, at least one composition-purposeimage that is same as at least one composition-purpose image combinedwith the previous image may be arranged so as to have a same positionalrelation with the at least one subject in the current captured image asa positional relation between the at least one subject in the previouscaptured image and the at least one composition-purpose image combinedwith the previous captured image.

[7] In the image processing apparatus of any one of [1] to [5], when theimage composition unit generates the composite image from each ofcontinuously captured images, until a predetermined time has passedsince an arbitrary image of the continuously captured images is capturedor until a predetermined number of captured images has been processed,at least one composition-purpose image that is same as at least onecomposition-purpose image combined with the previous image may bearranged so as to have a same positional relation with the at least onesubject in the current captured image as a positional relation betweenthe at least one subject in the previous captured image and the at leastone composition-purpose image combined with the previous captured image.

[8] In the image processing apparatus of [1], each subject may be ahuman face.

[9] According to another aspect of the invention, an imaging apparatusincludes the image processing apparatus of any one of [1] to [9].

[10] According to further another aspect of the invention, an imageprocessing method includes: detecting at least one subject in a capturedimage; judging arrangement of the at least one subject in the capturedimage by an arithmetic operation; selecting at least onecomposition-purpose image at random in accordance with the arrangementof the at least one subject; determining arrangement of the selected atleast one composition-purpose image at random so that the at least onecomposition-purpose image does not overlap with one another and does notoverlap with the detected at least one subject; and generating acomposite image by combining the captured image with the at least onecomposition-purpose image.

[11] The image processing method of [10] may further include selecting asize for each of the at least one composition-purpose image to bearranged, at random from an image size range that is set in advance.

[12] The image processing method of any one of [10] to [11] may furtherinclude selecting an inclination for each of the at least onecomposition-purpose images to be arranged, at random from an imageinclination range that is set in advance.

[13] In the image processing method of any one of [10] to [12], when thenumber of detected subject is two or more, the at least onecomposition-purpose image may be selected based on an arrangementrelation between the detected subjects.

[14] In the image processing method of [13], the at least onecomposition-purpose image may be selected based on each distance betweenthe subjects.

[15] In the image processing method of any one of [10] to [14], when thecomposite image is generated from each of continuously captured images,if arrangement of the at least one subject in a current captured imageis same as that of the at least one subject in a previous capturedimage, at least one composition-purpose image that is same as at leastone composition-purpose image combined with the previous image may bearranged so as to have a same positional relation with the at least onesubject in the current captured image as a positional relation betweenthe at least one subject in the previous captured image and the at leastone composition-purpose image combined with the previous captured image.

[16] In the image processing method of any one of [10] to [14], when thecomposite image is generated from each of continuously captured images,until a predetermined time has passed since an arbitrary image of thecontinuously captured images is captured or until a predetermined numberof captured images has been processed, at least one composition-purposeimage that is same as at least one composition-purpose image combinedwith the previous image may be arranged so as to have a same positionalrelation with the at least one subject in the current captured image asa positional relation between the at least one subject in the previouscaptured image and the at least one composition-purpose image combinedwith the previous captured image.

[17] In the image processing method of any one of [10] to [16], eachsubject may be a human face.

With the above configurations, while the arrangement of the at least onesubject is judged and the at least one composition-purpose image usedfor image composition is selected at random based on informationrelating to the arrangement of the at least one subject, the selected atleast one composition-purpose image can be arranged at random in anon-overlap manner in accordance with the arrangement of the at leastone subject. There is a possibility that composition-purpose image(s)will overlap with the subject(s) when the composition-purpose image(s)is arranged at complete random relative to the captured image. With theabove configurations, it is however possible to avoid that thesubject(s) overlap with the composition-purpose image(s) because thearrangement of the composition-purpose image(s) is determined based onthe arrangement of the subject(s). As a result, a composite image as acombination of the captured image and the composition-purpose images isdisplayed so as to be adapted to the scene of the captured image.

Moreover, a composite image giving surprise to a user can be generatedat the time of image capturing because the composition-purpose image(s)are selected at random and arranged in accordance with the arrangementof the subject(s).

In addition, when image composition is applied to images that arecaptured continuously, if arrangement of the subject(s) in the currentimage is same as that of the subject(s) in the previous image, thecomposition-purpose image(s) that are same as the composition-purposeimage(s) combined with the previous captured image can be arranged so asto have the same positional relation with the subject(s) in the currentcaptured image as the positional relation between the subject(s) in theprevious captured image and the composition-purpose image(s) combinedwith the previous captured image. Consequently, for example, display ofa through image or a motion image becomes easy to view.

According to the invention, it is possible to provide an imageprocessing apparatus, an image processing method and an imagingapparatus capable of generating a composite image matched with alocation scene by combining a captured image with a composition-purposeimage(s) based on at least one subject in the captured image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of the configuration of animage processing apparatus and an imaging apparatus according toembodiments of the invention.

FIG. 2 is a block diagram showing control for combining a captured imagewith a composition-purpose image(s).

FIG. 3 is a flow chart showing a processing procedure according to afirst embodiment of the invention.

FIGS. 4A and 4B are views each showing a state where detection of a faceposition is performed.

FIGS. 5A and 5B are views each showing a state where judgment of theface position is performed.

FIGS. 6A and 6B are views each showing an example of composition-purposeimages.

FIGS. 7A and 7B are views each showing a state where a composition areais set in a captured image.

FIGS. 8A and 8B are views each showing a state where composition-purposeimages are arranged in the composition area.

FIG. 9 is a view showing a state where a size and inclination of eachcomposition-purpose image are set.

FIGS. 10A and 10B are views each showing an example of the compositeimage generated by the processing according to the first embodiment.

FIG. 11 is a view showing a composite image generated without executionof the processing according to the first embodiment.

FIG. 12 is a view showing another example of the composite imagegenerated by the processing according to the first embodiment.

FIG. 13 is a flow chart showing a processing procedure according to asecond embodiment of the invention.

FIGS. 14A and 14B are views each showing a state where judgment relatingto a positional relation between subjects is performed.

FIG. 15 is a view showing examples of the composition-purpose images.

FIGS. 16A and 16B are views each showing a state where a compositionarea is set in a captured image.

FIGS. 17A and 17B are views each showing a state where eachcomposition-purpose image is arranged in a composition area.

FIGS. 18A and 18B are views each showing an example of the compositeimage generated by the processing according to the second embodiment.

FIG. 19 is a view showing another example of the composite imagegenerated by the processing according to the second embodiment.

FIG. 20 is a flow chart showing a processing procedure according to athird embodiment of the invention.

FIGS. 21A and 21B are views for explaining a state where eachcomposition-purpose image follows the subject in continuous frames.

FIG. 22 is a flow chart showing a processing procedure according to afourth embodiment of the invention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Embodiments of the invention will be described below in detail withreference to the drawings.

FIG. 1 is a block diagram showing an example of the configuration of animage processing apparatus and an imaging apparatus according to theembodiments of the invention. Although description in the followingembodiments will be made with reference to the configuration of adigital camera by way of example, the image processing apparatus and theimaging apparatus according to the invention are not limited theretoparticularly.

A digital camera 1 has an image capturing portion which captures animage of a subject. The image capturing portion has a lens 11, anaperture diaphragm 12, a filter 13, a solid-state imaging device (CCD)14, an analog signal processing portion 15, and an A/D conversionportion 16. The solid-state imaging device 14 generates image data inaccordance with incident light. The image data generated as an analogsignal by the solid-state imaging device 14 is input to the analogsignal processing portion 15. The A/D conversion portion 16 converts theprocessed analog signal into a digital signal. The image capturingportion further has a lens drive portion 17, an aperture diaphragm driveportion 18, and a CCD drive portion 19. The lens drive portion 17 drivesthe lens 11. The aperture diaphragm drive portion 18 drives the aperturediaphragm 12. The CCD drive portion 19 supplies a drive signal to thesolid-state imaging device 14. The lens drive portion 17, the aperturediaphragm drive portion 18 and the CCD drive portion 19 are connected toa CPU 20. Although a configuration using a CCD as the imaging device isshown in this embodiment, the imaging device need not be particularlylimited and a CMOS image sensor may be used as the imaging device.

An operation portion 21 that can be operated by a user at the time ofimage capturing is provided in the digital camera 1. The operationportion 21 is connected to the CPU 20.

Further, in the digital camera 1, a memory control portion 22, a digitalsignal processing portion 24, an expansion/compression portion 25, anintegrating portion 26, an external memory control portion 27, a displaycontrol portion 29 and an image processing portion 31 are connected tothe A/D conversion portion 16 and the CPU 20 via a bus. A main memory 23is connected to the memory control portion 22. An external memory 28 isconnected to the external memory control portion 27. A display portion30 such as an LCD is connected to the display control portion 29 so thata captured image or a composite image generated by combining thecaptured image with composition-purpose images can be displayed.

The digital camera 1 according to this embodiment further has a facedetection portion 42, a face position analysis portion 44, an imageselection portion 46, an image arrangement portion 48, and an imagecomposition portion 50. The face detection portion 42 serves as asubject detection unit that detects at least one predetermined subjects(e.g. human faces in this embodiment) in a captured image. The faceposition analysis portion 44 serves as a subject position analysis unitthat judges arrangement of the subject(s) by an arithmetic operation.The image selection portion 46 serves as an image selection unit thatselects a composition-purpose image(s) at random in accordance with thearrangement of the subject(s). The image arrangement portion 48 servesas an image arrangement unit that determines arrangement of the selectedcomposition-purpose image(s) at random so that the composition-purposeimages do not overlap with one another and do not overlap with thesubject(s). The image composition portion 50 serves as an imagecomposition unit that generates a composite image by combining thecaptured image with the composition-purpose image(s). The digital camera1 according to this embodiment further has a composition-purpose imagestorage portion 52 that stores composition-purpose images in advance.

FIG. 2 shows a block diagram showing control for combining a capturedimage with a composition-purpose image(s). Information of a capturedimage (which may be a through image as shown in FIG. 2 or may be a stillimage) is generated by the CCD 14 at the time of image capturing. Thecaptured image information is input both to the face detection portion42 and to the image composition portion 50. The face detection portion42 detects at least one desired subject from the captured imageinformation. Although each subject detected in this embodiment is ahuman face, the subject is not limited thereto, and another materialbody or a whole human figure may be used as the subject. Information ofa face position(s) detected by the face detection portion 42 is fed bothto the face position analysis portion 44 and to the image arrangementportion 48.

The face position analysis portion 44 performs an arithmetic operationbased on the face position information as to how the face position(s) is(are) arranged in the captured image, and feeds a result of the faceposition analysis to the image selection portion 46.

The image selection portion 46 selects a composition-purpose image(s) atrandom based on the result of the face position analysis. On thisoccasion, the composition-purpose image(s) are read from thecomposition-purpose image storage portion 52, which storescomposition-purpose images that are set in accordance with arrangementof face positions in advance. The image selection portion 46 feeds imageselection information to the image arrangement portion 48. The imagearrangement portion 48 determines arrangement of the selectedcomposition-purpose image(s) at random based on the face positioninformation, the image selection information and the composition-purposeimage(s) provided from the composition-purpose image storage portion 52so that the selected composition-purpose image(s) do not overlap withone another and do not overlap with the subject face(s). The imagearrangement portion 48 feeds the determined arrangement of thecomposition-purpose image(s) as image arrangement information to theimage composition portion 50.

The image composition portion 50 generates a composite image bycombining the captured image with the composition-purpose image(s) basedon the captured image provided from the CCD, the image arrangementinformation provided from the image arrangement portion 48 and thecomposition-purpose image(s) provided from the composition-purpose imagestorage portion 52. The image composition portion 50 outputs thecomposite image to the display portion 30, so that the composite imageis displayed on the display portion 30.

A procedure of image processing according to the embodiments of theinvention will be described below with reference to the drawings. FIG. 3is a flow chart showing the procedure of image processing according to afirst embodiment of the invention.

First, a face position is detected from an acquired captured image. Eachof FIGS. 4A and 4B shows a state where detection of a face position isperformed. FIG. 4A shows a state where the face position is in a centralportion of the captured image. FIG. 4B shows a state where the faceposition is in a peripheral portion of the captured image. Incidentally,in each of FIGS. 4A and 4B, the face position is shown as a regionsurrounded by the broken line. After detection of the face position,judgment is made as to whether the face position is in the center ornot.

Each of FIGS. 5A and 5B shows a state where the face position is judged.FIG. 5A shows a state where the face position is in the center of thecaptured image. FIG. 5B shows a state where the face position is in theperiphery of the captured image. Incidentally, in each of FIGS. 5A and5B, the face position is shown as a region surrounded by the brokenline.

As a specific example of the face position judgment, a peripheryjudgment region having a predetermined width W from the peripheral endof the captured image is set as shown in each of FIGS. 5A and 5B, sothat the face position overlapping with the periphery judgment regioncan be judged to be in the periphery while the face position notoverlapping with the periphery judgment region can be judged to be inthe center.

Then, zero or more appropriate images are selected at random fromcomposition-purpose images that are set in advance, based on the judgedface position. FIG. 6A is a view showing an example ofcomposition-purpose images. FIG. 6B is a view showing the number ofrepetitions of each selected composition-purpose image. ‘Zero or more’means that there may be the case where no composition-purpose image isselected, that is, the case where image composition is not performed. Asshown in FIG. 6A, composition-purpose images are prepared in a statewhere the composition-purpose images are separated into an image groupfor the center and an image group for the periphery. Zero or moreappropriate images are selected at random based on the face positionjudgment. Images captured in advance or images acquired from the outsidein advance may be used as the composition-purpose images. In selectionof the composition-purpose image(s), one composition-purpose image maybe selected repeatedly as shown in FIG. 6B or no composition-purposeimage may be selected. In addition, the user may set the maximum limitof the number of selected composition-purpose images in advance so thatcomposition-purpose images not larger than the maximum limit areselected.

After selection of the composition-purpose image(s), an appropriatecomposition area is set based on the face position judgment. Each ofFIGS. 7A and 7B shows a state where a composition area is set in thecaptured image. FIG. 7A shows a state where a composition area is setwhen the face position is in the center of the captured image. FIG. 7Bshows a state where a composition area is set when the face position isin the periphery of the captured image. As shown in FIGS. 7A and 7B, inthis embodiment, the position and shape of the composition area are setin accordance with the face position. Although the shape of thecomposition area is arbitrary, a region not overlapping with the subject(face position) is set as the composition area in this embodiment.

Successively, a composition-purpose image(s) are arranged one by one atrandom in the set composition area. Each of FIGS. 8A and 8B shows astate where composition-purpose images are arranged in the compositionarea. FIG. 8A shows a state where composition-purpose images arearranged in the composition area when the face position is in the centerof the captured image. FIG. 8B shows a state where composition-purposeimages are arranged in the composition area when the face position is inthe periphery of the captured image. The position, size and direction ofeach composition-purpose image are determined at random but thearrangement of a composition-purpose image(s) is selected so that thecomposition-purpose image(s) are disposed in the composition area and donot to overlap with one another.

FIG. 9 shows a state where the size and inclination are set for eachcomposition-purpose image. Each of FIGS. 10A and 10B shows a state wherecomposition-purpose images are combined with a captured image. FIG. 10Ashows a state where composition-purpose images are combined with acaptured image when the face position is in the center of the capturedimage. FIG. 10B shows a state where composition-purpose images arecombined with a captured image when the face position is in theperiphery of the captured image. As shown in FIG. 9, a size range offrom a ‘minimum size’ to a ‘maximum size’ is defined for eachcomposition-purpose image so that the size of each composition-purposeimage can be selected at random from the size range. In this manner, thesize of each composition-purpose image can be prevented from becomingtoo small to view or too large unnaturally after eachcomposition-purpose image is combined with the captured image. Withrespect to the direction of each composition-purpose image, aninclination range (e.g. from −30° to +30°) of each composition-purposeimage with respect to the captured image is defined in the same manneras described above so that each composition-purpose image is preventedfrom being arranged in an unnatural direction relative to a scene of thecaptured image.

When all the predetermined composition-purpose images are arranged and acomposite image is generated by combining the captured image with allthe arranged composition-purpose images, processing according to thisembodiment is terminated. Execution of the aforementioned imageprocessing permits appropriate composition-purpose images to be combinedwith the captured image in appropriate arrangement in accordance withthe arrangement of the subject.

FIG. 11 is a view showing a composite image generated without executionof the processing according to this embodiment. This embodiment canavoid that the composition-purpose images overlap with the subject as inthe case where the composition-purpose images are arranged completely atrandom relative to the captured image as shown in FIG. 11. This isbecause the composition-purpose images to be combined with the capturedimage are selected at random based on the subject arrangementinformation obtained by judgment of the arrangement of the subject(face) and are arranged at random in accordance with the arrangement ofthe subject so as not to overlap with one another and with the subject.According to this embodiment, the subject arrangement information isused for selection of the composition-purpose images and fordetermination of composition positions of the composition-purpose imagesso that the composition-purpose images are selected and arranged notcompletely at random but with certain qualifications based on thearrangement of the subject. Accordingly, because the arrangement of thecomposition-purpose images is determined based on the arrangement of thesubject, it can be avoided that the composition-purpose images overlapwith the subject. Therefore, the resulting composite image can bedisplayed so as to be adapted to a scene of the captured image.

In addition, when the face position protrudes from a frame of thecaptured image as shown in FIG. 10B, a display different from that inthe case where the face position is in the center (see FIG. 10A) can bemade to call attention to the user.

FIG. 12 is a view showing another example of a composite image generatedby the processing according to this embodiment. Because the selectedcomposition-purpose images are random in terms of arrangement, numberand type, the resulting composite image varies whenever an unchangedscene is captured. For example, even if the same scene as in FIG. 10A inwhich the face position is in the center is captured again in preciselythe same condition, there is a possibility that a composite image havingarrangement of composition-purpose images different from those in thecomposite image shown in FIG. 10A may be obtained as shown in FIG. 12.In this manner, the resulting composite image varies for respectiveimage capturings, so that the user can enjoy unexpected image capturing.

Incidentally, this embodiment is only exemplary and can be modifiedappropriately. For example, the same processing may be applied toanother subject than a face. Although this embodiment has been describedon the case where only a position of a face in the arrangement of thesubject is used as a criterion for selection of a composition-purposeimage(s) and definition of a composition area, the size and direction ofthe face may be used as a criterion for selection of thecomposition-purpose images and definition of the composition area.Although this embodiment has been described on the case where thecomposition area is defined before the composition-purpose image(s) arearranged in the composition area, the same effect can be obtained alsoin the case where judgment is made as to whether the arrangement of thecomposition-purpose image(s) is appropriate after thecomposition-purpose image(s) are arranged without definition of thecomposition area so that the composition-purpose images are rearrangedwhen the arrangement of the composition-purpose image(s) is notappropriate. Accordingly, the procedure of defining the composition areain this embodiment is not an essential process.

A second embodiment of the invention will be described below withreference to the drawings.

FIG. 13 is a flow chart showing a processing procedure according to thisembodiment. In this embodiment, processing is performed while not onlyarrangement of one subject but also a relation between subjects is takeninto consideration.

First, positions of faces and the number of the faces are detected froma captured image. Each of FIGS. 14A and 14B shows a state where twofaces are in a captured image. FIG. 14A shows a state where the facesare close to each other. FIG. 14B shows a state where the faces are farfrom each other. Then, judgment is made as to whether the number offaces is two or not. If the number of faces is not equal to two,processing is terminated. If the number of faces is equal to two,judgment is further made as to whether or not a distance between thefaces is smaller than a threshold TH. For example, the threshold TH is avalue that is determined based on the sizes of the two faces. Forexample, an average of the widths of the two faces may be defined as thethreshold TH.

Then, zero or more appropriate images are selected at random fromcomposition-purpose images that are set in advance, based on a result ofthe judgment as for the distance between the faces. FIG. 15 is a viewshowing an example of composition-purpose images. As shown in FIG. 15,composition-purpose images for the close distance between the faces andcomposition-purpose images for the far distance between the faces areprepared separately in advance. Zero or more images are selected atrandom from appropriate images in accordance with the face distancejudgment. Incidentally, in this embodiment, one composition-purposeimage may be selected repeatedly or no composition-purpose image may beselected, similarly to the aforementioned embodiment. In addition, themaximum number of repetitions of each composition-purpose image may beset in advance so that each composition-purpose image can be selected sorepeatedly that the number of repetitions of each image does not exceedthe maximum number.

Then, an appropriate composition area is set based on the face distancejudgment. Each of FIGS. 16A and 16B is a view showing a state where acomposition area is set in the captured image. FIG. 16A shows a statewhere a composition area is set when the faces are close to each other.FIG. 16B shows a state where a composition area is set when the facesare far from each other. Incidentally, the shape of the composition areais arbitrary but a region not overlapping with subjects (face regions)is set as the composition area in this embodiment.

Successively, the composition-purpose image(s) are arranged one by oneat random in the composition area. Each of FIGS. 17A and 17B is a viewshowing a state where each composition-purpose image is arranged in thecomposition area. FIG. 17A shows a state where the composition-purposeimages are arranged in the composition area when the faces are close toeach other. FIG. 17B shows a state where the composition-purpose imageis arranged in the composition area when the faces are far from eachother. The positions, sizes and directions of the composition-purposeimages are arbitrary but arrangement of the composition-purpose imagesis selected so that the composition-purpose images are put in thecomposition area and do not overlap with one another. As shown in FIG.9, a size range of from a ‘minimum size’ to a ‘maximum size’ is definedfor each composition-purpose image. The size of each composition-purposeimage is selected at random from the size range so that eachcomposition-purpose image can be prevented from becoming too small toview or too large unnaturally. With respect to the direction of eachcomposition-purpose image, an inclination range of eachcomposition-purpose image with the captured image is defined in the samemanner as described above so that each composition-purpose image havingan orientation can be prevented from being arranged in an unnaturaldirection. When a composite image is generated by combining the capturedimage with all the arranged composition-purpose images, the processingaccording to this embodiment is terminated.

Each of FIGS. 18A and 18B is a view showing an example of the compositeimage generated by the processing according to this embodiment. FIG. 18Ashows a state where the composition-purpose images are combined with thecaptured image when the faces are close to each other. FIG. 183 shows astate where the composition-purpose image is combined with the capturedimage when the faces are far from each other.

In this embodiment, the captured image is combined with each appropriatecomposition-purpose image arranged appropriately in accordance with therelation in arrangement between the subjects. Accordingly, because thecaptured image is combined with each of the predeterminedcomposition-purpose images, which are individually provided for (i) thecase where the faces are close to each other (FIG. 18A) and (ii) thecase where the faces are far from each other (FIG. 18B), display havingdifference visual effects can be made. Moreover, because each selectedcomposition-purpose image is random in terms of arrangement, number andtype, the resulting composite image varies whenever an unchanged sceneis captured. For example, even if the same scene as in FIG. 18B in whichthe faces are far from each other is captured again in the samecondition, there is a possibility that a composite image as shown inFIG. 19 may be obtained. In this manner, the different composite imagesare generated for respective image capturings, so that the user canenjoy unexpected image capturing.

Incidentally, the procedure in this embodiment is only exemplary and,for example, the same processing may be applied to other subjects thanfaces. Although this embodiment has been described on the case whereonly the relation in position between the faces in the arrangement ofthe subjects is used as a criterion for selection of thecomposition-purpose images and definition of the composition area, therelation in size and direction between the faces may be used as acriterion for selection of the composition-purpose images and definitionof the composition area. Although this embodiment has been described onthe case where the composition area is defined before thecomposition-purpose images are arranged in the composition area, thesame effect as that in this embodiment can be obtained also in the casewhere judgment is made as to whether the arrangement of thecomposition-purpose images is appropriate after the composition-purposeimages are arranged without definition of the composition area so thatthe composition-purpose images are rearranged when the arrangement ofthe composition-purpose images is not appropriate. Accordingly, theprocedure of defining the composition area in this embodiment is not anessential process.

A third embodiment of the invention will be described below withreference to the drawings. FIG. 20 is a flow chart showing a processingprocedure according to this embodiment. An example of compositionprocessing that is applied to continuously captured images will bedescribed in this embodiment.

As shown in FIG. 20, the step of detecting a face position, the step ofjudging the arrangement of the face position by an arithmetic operation,the step of selecting a composition-purpose image(s) at random inaccordance with the arrangement of the subject and the step ofdetermining the arrangement of the selected composition-purpose image(s)at random so that the composition-purpose image(s) do not overlap withone another and do not overlap with the subject are the same as in thefirst embodiment.

In this embodiment, when a composition-purpose image(s) are combinedwith continuously captured images by referring to the arrangement of thesubject in a previous frame. If the arrangement of the subject isunchanged, the same composition-purpose image(s) as those are used forthe previous frame are arranged and combined to obtain the samepositional relation as a positional relation between the subject in theprevious frame and the composition-purpose image(s) used for theprevious frame.

FIGS. 21A and 21B are views for explaining continuously captured images.FIG. 21A shows a captured image in a previous frame. FIG. 21B shows acaptured image in a current frame following the captured image shown inFIG. 21A. The current frame to be processed is referred to as t-th framewhile the previous frame is referred to as (t−1)th frame. As shown inFIG. 20, after the face position is analyzed to judge as to whether theface position is in the center or in the periphery of the capturedimage, the face position in the (t−1)th frame is referred to. If theface position in the (t−1)th frame coincides with the face position inthe t frame, a ‘process A’ is performed. When the face position in thet−1 frame does not coincide with the face position in the t-th frame,the same process as in the first embodiment is performed. Incidentally,the face position in the first frame of continuously captured images isalways judged to be not coincident with that in the previous framebecause there is no frame previous to the first frame.

In the process A, the same composition-purpose image(s) as those usedfor generating the composite image of the (t−1)-th frame are arranged toobtain a positional relation equivalent to a positional relation betweenthe face in the (t−1)th frame and the composition-purpose image(s) inthe (t−1)th frame. Specifically, the coordinates (x_(t), y_(t)) of eachcomposition-purpose image in the t-th frame is calculated by thefollowing expression (1) using the face position (fx_(t), fy_(t)), thecomposition position (x_(t-1), y_(t-1)) and the face position (fx_(t-1),fy_(t-1)) in the (t−1)th frame.x _(t) =fx _(t)+(x _(t-1) −fx _(t-1))y _(t) =fy _(t)+(y _(t-1) −fy _(t-1))  (1)

As shown in FIGS. 21A and 21B, the positional relation between the faceand the composition-purpose image in the t-th frame is equivalent to thepositional relation between the face and the composition-purpose imagein the (t−1)th frame. Incidentally, the position of the face and theposition of the composition-purpose image are expressed by thecoordinates of left upper points of respective regions of the face andthe composition-purpose image. Because the positional relation is keptconstant, the composition-purpose image moves following the subject evenif the subject moves between the frames.

After a composite image is generated, the face position and the positionof each composition-purpose image are recorded on a recording portion orthe like. This is because the face position and the position of eachcomposition-purpose image are required for processing a next frame.

Incidentally, this embodiment is only exemplary and can be modifiedappropriately. For example, it is thought of that the same processing isapplied to another subject than the face. Although this embodiment hasbeen described on the case where processing is performed so that onlythe positional relation between the face as the arrangement of thesubject and each composition-purpose image is kept constant, it ispreferable that the size and direction of each composition-purpose imageare processed in the same manner as described above. Moreover, whenthere are plural subjects, the relation between the subjects may beanalyzed in the same processing as described above based on judgment asto whether or not the relation in the current frame is equivalent to therelation in the previous frame.

Although this embodiment has been described on the case where theabsolute position of the subject and the absolute position of eachcomposition-purpose image are recorded so as to be referred to forprocessing the next frame, the same effect as described above can beobtained also in the case where a result of analysis of the arrangementof the subject and the position of each composition-purpose imagerelative to the subject are used.

A fourth embodiment of the invention will be described below withreference to the drawings.

FIG. 22 is a flow chart showing a processing procedure according to thisembodiment. This embodiment will be described by way of example on thecase where a composition process is applied to one frame image whencomposition-purpose images are combined with continuously capturedimages. Incidentally, the step of detecting a face position, the step ofjudging the arrangement of the face position by an arithmetic operation,the step of selecting composition-purpose images at random in accordancewith the arrangement of the subject and the step of determining thearrangement of the selected composition-purpose images at random so thatthe composition-purpose images do not overlap with one another and donot overlap with the subject are the same as in the first embodiment.

In this embodiment, when composition-purpose images are combined withcontinuously captured images, the same composition-purpose image(s) asthose used for the previous frame are arranged to obtain the samepositional relation as the positional relation between the subject andeach composition-purpose image in the previous frame until apredetermined time has passed or until a specific number of frames havebeen processed.

As shown in FIG. 22, a ‘process A’ is performed in the same manner as inthe third embodiment if a value I of a counter indicating the number ofprocessed frames is smaller than a constant α. When the value I of thecounter indicating the number of processed frames is not smaller thanthe constant α, the value I of the counter indicating the number ofprocessed frames is initialized to zero and the process described in thefirst embodiment is performed. When, for example, the constant α is setat 30, the composition process as described in the first embodiment isperformed at intervals of 30 frames so that the resulting compositeimage changes. On the other hand, the ‘process A’ is applied to a frameat I smaller than 30, so that each composition-purpose image movesfollowing the subject. In this embodiment, after the composition processis executed, the value I of the counter indicating the number ofprocessed frames is updated (I=I+1 in FIG. 22) to make preparation forprocessing the next frame.

If the composition process described in the first embodiment is executedfor each frame of the continuously captured images (through images ormotion images), a result of composition varies widely according to theframe so that the resulting composite images are of no practical usebecause they are very hard to view due to flickering. Therefore, inaccordance with this embodiment, the resulting composite images arestabilized because the composite image is updated at regular intervalsso that each composition-purpose image follows the subject until thecomposite image is updated. Moreover, whenever the arrangement of thesubject changes or whenever image capturing is started again, adifferent composition result can be obtained by random image compositionso that the user can enjoy unexpected image capturing.

What is claimed is:
 1. An image processing apparatus comprising: asubject detection unit that detects at least one subject in a capturedimage; a subject position analysis unit that judges arrangement of theat least one subject in the captured image by an arithmetic operation;an image selection unit that selects at least one composition-purposeimage in accordance with the arrangement of the at least one subject; animage arrangement unit that determines arrangement of the selected atleast one composition-purpose image so that the at least onecomposition-purpose image does not overlap with one another and does notoverlap with the detected at least one subject; and an image compositionunit that generates a composite image by combining the captured imagewith the at least one composition-purpose image, wherein when the numberof detected subject is two or more, the image arrangement unit selectsthe at least one composition-purpose image based on an arrangementrelation between the detected subjects.
 2. The image processingapparatus according to claim 1, wherein the image arrangement unitselects a size for each of the at least one composition-purpose image tobe arranged, at random from an image size range that is set in advance.3. The image processing apparatus according to claim 1, wherein theimage arrangement unit selects an inclination for each of the at leastone composition-purpose images to be arranged, at random from an imageinclination range that is set in advance.
 4. The image processingapparatus according to claim 1, wherein the image selection unit selectsthe at least one composition-purpose image based on each distancebetween the subjects.
 5. The image processing apparatus according toclaim 1, wherein in a case that (i) the image composition unit generatesthe composition-purpose images for each captured image which iscontinuously captured, (ii) first and second captured images arecaptured in order, (iii) the at least one subject in the first capturedimage includes a first subject and the at least one subject in thesecond captured image includes a second subject and (iv) the arrangementof the second subject is the same arrangement of the first subject, atleast a first composition-purpose image is generated by combined withthe first captured image, at least one composition-purpose image beingthe same image of the first composition-purpose image is arranged so asto have a same position relationship with the second subject, theposition relationship being between the first subject and said firstcomposition-purpose image.
 6. The image processing apparatus accordingto claim 1, wherein in a case that (i) the image composition unitgenerates the composition-purpose images for each captured image whichis continuously captured, (ii) first and second captured images arecaptured in order, and (iii) the at least one subject in the firstcaptured image includes a first subject and the at least one subject inthe second captured image includes a second subject, at least a firstcomposition-purpose image is generated by combined with the firstcaptured image, and (iv) until a predetermined time is elapsed since afirst captured image is captured or (v) until a predetermined number ofthe captured images is processed, at least one composition-purpose imagebeing the same image of the first composition-purpose image is arrangedso as to have a same position relationship with the second subject, theposition relationship being between the first subject and said firstcomposition-purpose image.
 7. The image processing apparatus accordingto claim 1, wherein each subject is a human face.
 8. The imageprocessing apparatus according to claim 1, wherein the image selectionunit selects at least one composition-purpose image without a randomselecting, and the image arrangement unit determines the arrangement ofthe selected at least one composition-purpose image without a randomselecting.
 9. An imaging apparatus comprising: the image processingapparatus according to claim
 1. 10. An image processing methodcomprising: detecting at least one subject in a captured image; judgingarrangement of the at least one subject in the captured image by anarithmetic operation; selecting at least one composition-purpose imagein accordance with the arrangement of the at least one subject;determining arrangement of the selected at least one composition-purposeimage so that the at least one composition-purpose image does notoverlap with one another and does not overlap with the detected at leastone subject; and generating a composite image by combining the capturedimage with the at least one composition-purpose image wherein when thenumber of detected subject is two or more, the at least onecomposition-purpose image is selected based on an arrangement relationbetween the detected subjects.